Design and Evaluation of an American Sign Language Generator

We describe the implementation and evaluation of a prototype American Sign Language (ASL) generation component that produces animations of ASL classifier predicates, some frequent and complex spatial phenomena in ASL that no previous generation system has produced. We discuss some challenges in evaluating ASL systems and present the results of a userbased evaluation study of our system. 1 Background and Motivations American Sign Language (ASL) is a natural language with a linguistic structure distinct from English used as a primary means of communication for approximately one half million people in the U.S. (Mitchell et al., 2006). A majority of deaf 18-yearolds in the U.S. have an English reading level below that of an average 10-year-old hearing student (Holt, 1991), and so software to translate English text into ASL animations can improve many people’s access to information, communication, and services. Previous English-to-ASL machine translation projects (Sáfár & Marshall, 2001; Zhou et al., 2000) could not generate classifier predicates (CPs), phenomena in which signers use special hand movements to indicate the location and movement of invisible objects in space around them (representing entities under discussion). Because CPs are frequent in ASL and necessary for conveying many concepts, we have developed a CP generator that can be incorporated into a full English-to-ASL machine translation system. During a CP, signers use their hands to position, move, trace, or re-orient imaginary objects in the space in front of them to indicate the location, movement, shape, contour, physical dimension, or some other property of corresponding real world entities under discussion. CPs consist of a semantically meaningful handshape and a 3D hand movement path. A handshape is chosen from a closed set based on characteristics of the entity described (whether it be a vehicle, human, animal, etc.) and what aspect of the entity the signer is describing (surface, position, motion, etc). For example, the sentence “the car parked between the cat and the house” could be expressed in ASL using 3 CPs. First, a signer performs the ASL sign HOUSE while raising her eyebrows (to introduce a new entity as a topic). Then, she moves her hand in a “Spread C” handshape (Figure 1) forward to a point in space where a miniature house could be envisioned. Next, the signer performs the sign CAT with eyebrows raised and makes a similar motion with a “Hooked V” handshape to a location where a cat could be imagined. Finally, she performs the sign CAR (with eyebrows raised) and uses a “Number 3” handshape to trace a path that stops at between the ‘house’ and the ‘cat.’ Her other hand makes a flat surface for the ‘car’ to park on. (Figure 3 will show our system’s animation.) Figure 1: ASL handshapes: Spread C (bulky object), Number 3 (vehicle), Hooked V (animal), Flat (surface).